from google.colab import drive
drive.mount("/content/drive")
Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount("/content/drive", force_remount=True).
!pip install h2o
Requirement already satisfied: h2o in /usr/local/lib/python3.7/dist-packages (3.34.0.3) Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from h2o) (2.23.0) Requirement already satisfied: future in /usr/local/lib/python3.7/dist-packages (from h2o) (0.16.0) Requirement already satisfied: tabulate in /usr/local/lib/python3.7/dist-packages (from h2o) (0.8.9) Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->h2o) (3.0.4) Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->h2o) (1.24.3) Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->h2o) (2.10) Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->h2o) (2021.5.30)
import h2o
from h2o.automl import H2OAutoML
h2o.init()
# Import data
f = "/content/drive/My Drive/bug_pred.csv"
df = h2o.import_file(f)
# Reponse column
y = "defects"
# Split into train & test
splits = df.split_frame(ratios = [0.8], seed = 1)
train = splits[0]
test = splits[1]
# Run AutoML for 1 minute
aml = H2OAutoML(max_runtime_secs=60, seed=1)
aml.train(y=y, training_frame=train)
# Explain leader model & compare with all AutoML models
exa = aml.explain(test)
# Explain a single H2O model (e.g. leader model from AutoML)
exm = aml.leader.explain(test)
# Explain a generic list of models
# use h2o.explain as follows:
# exl = h2o.explain(model_list, test)
# 1. loc : numeric % McCabe's line count of code
# 2. v(g) : numeric % McCabe "cyclomatic complexity"
# 3. ev(g) : numeric % McCabe "essential complexity"
# 4. iv(g) : numeric % McCabe "design complexity"
# 5. n : numeric % Halstead total operators + operands
# 6. v : numeric % Halstead "volume"
# 7. l : numeric % Halstead "program length"
# 8. d : numeric % Halstead "difficulty"
# 9. i : numeric % Halstead "intelligence"
# 10. e : numeric % Halstead "effort"
# 11. b : numeric % Halstead
# 12. t : numeric % Halstead's time estimator
# 13. lOCode : numeric % Halstead's line count
# 14. lOComment : numeric % Halstead's count of lines of comments
# 15. lOBlank : numeric % Halstead's count of blank lines
# 16. lOCodeAndComment: numeric
# 17. uniq_Op : numeric % unique operators
# 18. uniq_Opnd : numeric % unique operands
# 19. total_Op : numeric % total operators
# 20. total_Opnd : numeric % total operands
# 21: branchCount : numeric % of the flow graph
# 22. defects : {false,true} % module has/has not one or more
# % reported defects
Checking whether there is an H2O instance running at http://localhost:54321 . connected.
| H2O_cluster_uptime: | 2 mins 57 secs |
| H2O_cluster_timezone: | Etc/UTC |
| H2O_data_parsing_timezone: | UTC |
| H2O_cluster_version: | 3.34.0.3 |
| H2O_cluster_version_age: | 28 days, 23 hours and 22 minutes |
| H2O_cluster_name: | H2O_from_python_unknownUser_a1bber |
| H2O_cluster_total_nodes: | 1 |
| H2O_cluster_free_memory: | 3.162 Gb |
| H2O_cluster_total_cores: | 2 |
| H2O_cluster_allowed_cores: | 2 |
| H2O_cluster_status: | locked, healthy |
| H2O_connection_url: | http://localhost:54321 |
| H2O_connection_proxy: | {"http": null, "https": null} |
| H2O_internal_security: | False |
| H2O_API_Extensions: | Amazon S3, XGBoost, Algos, AutoML, Core V3, TargetEncoder, Core V4 |
| Python_version: | 3.7.12 final |
Parse progress: |████████████████████████████████████████████████████████████████| (done) 100% AutoML progress: |███████████████████████████████████████████████████████████████| (done) 100%
Leaderboard shows models with their metrics. When provided with H2OAutoML object, the leaderboard shows 5-fold cross-validated metrics by default (depending on the H2OAutoML settings), otherwise it shows metrics computed on the frame. At most 20 models are shown by default.
| model_id | auc | logloss | aucpr | mean_per_class_error | rmse | mse | training_time_ms | predict_time_per_row_ms | algo |
|---|---|---|---|---|---|---|---|---|---|
| StackedEnsemble_BestOfFamily_4_AutoML_2_20211105_110657 | 0.814457 | 0.245272 | 0.221993 | 0.224564 | 0.270861 | 0.0733656 | 447 | 0.053338 | StackedEnsemble |
| GBM_grid_1_AutoML_2_20211105_110657_model_3 | 0.811326 | 0.240765 | 0.276307 | 0.26003 | 0.263394 | 0.0693766 | 126 | 0.020772 | GBM |
| GBM_grid_1_AutoML_2_20211105_110657_model_5 | 0.806839 | 0.254764 | 0.212941 | 0.312257 | 0.273922 | 0.0750332 | 212 | 0.01491 | GBM |
| StackedEnsemble_BestOfFamily_3_AutoML_2_20211105_110657 | 0.805146 | 0.2482 | 0.198759 | 0.211021 | 0.272885 | 0.0744661 | 337 | 0.036874 | StackedEnsemble |
| StackedEnsemble_BestOfFamily_2_AutoML_2_20211105_110657 | 0.802184 | 0.240283 | 0.201232 | 0.246699 | 0.268407 | 0.0720426 | 343 | 0.028962 | StackedEnsemble |
| DRF_1_AutoML_2_20211105_110657 | 0.796259 | 0.262609 | 0.237707 | 0.242932 | 0.276589 | 0.0765014 | 146 | 0.015528 | DRF |
| DeepLearning_1_AutoML_2_20211105_110657 | 0.795835 | 0.392299 | 0.234921 | 0.300872 | 0.289422 | 0.083765 | 111 | 0.011175 | DeepLearning |
| StackedEnsemble_AllModels_1_AutoML_2_20211105_110657 | 0.795412 | 0.249444 | 0.194681 | 0.208651 | 0.272874 | 0.0744604 | 431 | 0.052121 | StackedEnsemble |
| StackedEnsemble_AllModels_2_AutoML_2_20211105_110657 | 0.794735 | 0.249774 | 0.19746 | 0.238742 | 0.274122 | 0.0751431 | 437 | 0.055642 | StackedEnsemble |
| XGBoost_grid_1_AutoML_2_20211105_110657_model_3 | 0.79118 | 0.262332 | 0.230944 | 0.309463 | 0.276295 | 0.0763388 | 217 | 0.014467 | XGBoost |
| GBM_3_AutoML_2_20211105_110657 | 0.790503 | 0.263489 | 0.196818 | 0.31505 | 0.276093 | 0.0762273 | 189 | 0.020172 | GBM |
| GBM_grid_1_AutoML_2_20211105_110657_model_1 | 0.790249 | 0.255074 | 0.239563 | 0.270019 | 0.271006 | 0.0734441 | 121 | 0.029482 | GBM |
| XGBoost_grid_1_AutoML_2_20211105_110657_model_7 | 0.789318 | 0.24896 | 0.210234 | 0.291519 | 0.265976 | 0.0707431 | 112 | 0.010807 | XGBoost |
| XGBoost_grid_1_AutoML_2_20211105_110657_model_8 | 0.788217 | 0.328012 | 0.194577 | 0.26604 | 0.294134 | 0.086515 | 138 | 0.012883 | XGBoost |
| XGBoost_3_AutoML_2_20211105_110657 | 0.780472 | 0.265368 | 0.206416 | 0.246699 | 0.276454 | 0.0764271 | 446 | 0.013474 | XGBoost |
| GBM_grid_1_AutoML_2_20211105_110657_model_4 | 0.780388 | 0.251617 | 0.238559 | 0.29173 | 0.267124 | 0.0713554 | 189 | 0.019175 | GBM |
| XGBoost_grid_1_AutoML_2_20211105_110657_model_10 | 0.77933 | 0.315134 | 0.178319 | 0.294735 | 0.29535 | 0.0872314 | 194 | 0.012541 | XGBoost |
| GBM_2_AutoML_2_20211105_110657 | 0.778399 | 0.261371 | 0.20568 | 0.295497 | 0.272749 | 0.0743919 | 191 | 0.02372 | GBM |
| XGBoost_grid_1_AutoML_2_20211105_110657_model_2 | 0.774843 | 0.257768 | 0.193585 | 0.273997 | 0.271995 | 0.0739815 | 159 | 0.011466 | XGBoost |
| XGBoost_grid_1_AutoML_2_20211105_110657_model_5 | 0.766675 | 0.258487 | 0.18156 | 0.311072 | 0.268888 | 0.0723009 | 116 | 0.009935 | XGBoost |
Confusion matrix shows a predicted class vs an actual class.
Confusion Matrix (Act/Pred) for max f1 @ threshold = 0.15450129064581739:
| false | true | Error | Rate | ||
|---|---|---|---|---|---|
| 0 | false | 76.0 | 15.0 | 0.1648 | (15.0/91.0) |
| 1 | true | 5.0 | 11.0 | 0.3125 | (5.0/16.0) |
| 2 | Total | 81.0 | 26.0 | 0.1869 | (20.0/107.0) |
The variable importance plot shows the relative importance of the most important variables in the model.
Variable importance heatmap shows variable importance across multiple models. Some models in H2O return variable importance for one-hot (binary indicator) encoded versions of categorical columns (e.g. Deep Learning, XGBoost). In order for the variable importance of categorical columns to be compared across all model types we compute a summarization of the the variable importance across all one-hot encoded features and return a single variable importance for the original categorical feature. By default, the models and variables are ordered by their similarity.
This plot shows the correlation between the predictions of the models. For classification, frequency of identical predictions is used. By default, models are ordered by their similarity (as computed by hierarchical clustering). Interpretable models, such as GAM, GLM, and RuleFit are highlighted using red colored text.
SHAP summary plot shows the contribution of the features for each instance (row of data). The sum of the feature contributions and the bias term is equal to the raw prediction of the model, i.e., prediction before applying inverse link function.
Partial dependence plot (PDP) gives a graphical depiction of the marginal effect of a variable on the response. The effect of a variable is measured in change in the mean response. PDP assumes independence between the feature for which is the PDP computed and the rest.
Confusion matrix shows a predicted class vs an actual class.
Confusion Matrix (Act/Pred) for max f1 @ threshold = 0.15450129064581739:
| false | true | Error | Rate | ||
|---|---|---|---|---|---|
| 0 | false | 76.0 | 15.0 | 0.1648 | (15.0/91.0) |
| 1 | true | 5.0 | 11.0 | 0.3125 | (5.0/16.0) |
| 2 | Total | 81.0 | 26.0 | 0.1869 | (20.0/107.0) |
Partial dependence plot (PDP) gives a graphical depiction of the marginal effect of a variable on the response. The effect of a variable is measured in change in the mean response. PDP assumes independence between the feature for which is the PDP computed and the rest.
import h2o
from h2o.automl import H2OAutoML
import matplotlib
import pandas
h2o.init()
Checking whether there is an H2O instance running at http://localhost:54321 . connected.
| H2O_cluster_uptime: | 1 hour 9 mins |
| H2O_cluster_timezone: | Asia/Kolkata |
| H2O_data_parsing_timezone: | UTC |
| H2O_cluster_version: | 3.36.0.3 |
| H2O_cluster_version_age: | 18 days |
| H2O_cluster_name: | H2O_from_python_Venkat_sm0uc4 |
| H2O_cluster_total_nodes: | 1 |
| H2O_cluster_free_memory: | 3.673 Gb |
| H2O_cluster_total_cores: | 8 |
| H2O_cluster_allowed_cores: | 8 |
| H2O_cluster_status: | locked, healthy |
| H2O_connection_url: | http://localhost:54321 |
| H2O_connection_proxy: | {"http": null, "https": null} |
| H2O_internal_security: | False |
| Python_version: | 3.10.2 final |
# Import data
f = "hour.csv"
df = pd.read_csv(f)
df.describe()
| instant | season | yr | mnth | hr | holiday | weekday | workingday | weathersit | temp | atemp | hum | windspeed | casual | registered | cnt | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| count | 17379.0000 | 17379.000000 | 17379.000000 | 17379.000000 | 17379.000000 | 17379.000000 | 17379.000000 | 17379.000000 | 17379.000000 | 17379.000000 | 17379.000000 | 17379.000000 | 17379.000000 | 17379.000000 | 17379.000000 | 17379.000000 |
| mean | 8690.0000 | 2.501640 | 0.502561 | 6.537775 | 11.546752 | 0.028770 | 3.003683 | 0.682721 | 1.425283 | 0.496987 | 0.475775 | 0.627229 | 0.190098 | 35.676218 | 153.786869 | 189.463088 |
| std | 5017.0295 | 1.106918 | 0.500008 | 3.438776 | 6.914405 | 0.167165 | 2.005771 | 0.465431 | 0.639357 | 0.192556 | 0.171850 | 0.192930 | 0.122340 | 49.305030 | 151.357286 | 181.387599 |
| min | 1.0000 | 1.000000 | 0.000000 | 1.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 1.000000 | 0.020000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 1.000000 |
| 25% | 4345.5000 | 2.000000 | 0.000000 | 4.000000 | 6.000000 | 0.000000 | 1.000000 | 0.000000 | 1.000000 | 0.340000 | 0.333300 | 0.480000 | 0.104500 | 4.000000 | 34.000000 | 40.000000 |
| 50% | 8690.0000 | 3.000000 | 1.000000 | 7.000000 | 12.000000 | 0.000000 | 3.000000 | 1.000000 | 1.000000 | 0.500000 | 0.484800 | 0.630000 | 0.194000 | 17.000000 | 115.000000 | 142.000000 |
| 75% | 13034.5000 | 3.000000 | 1.000000 | 10.000000 | 18.000000 | 0.000000 | 5.000000 | 1.000000 | 2.000000 | 0.660000 | 0.621200 | 0.780000 | 0.253700 | 48.000000 | 220.000000 | 281.000000 |
| max | 17379.0000 | 4.000000 | 1.000000 | 12.000000 | 23.000000 | 1.000000 | 6.000000 | 1.000000 | 4.000000 | 1.000000 | 1.000000 | 1.000000 | 0.850700 | 367.000000 | 886.000000 | 977.000000 |
df.info()
<class 'pandas.core.frame.DataFrame'> RangeIndex: 17379 entries, 0 to 17378 Data columns (total 17 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 instant 17379 non-null int64 1 dteday 17379 non-null object 2 season 17379 non-null int64 3 yr 17379 non-null int64 4 mnth 17379 non-null int64 5 hr 17379 non-null int64 6 holiday 17379 non-null int64 7 weekday 17379 non-null int64 8 workingday 17379 non-null int64 9 weathersit 17379 non-null int64 10 temp 17379 non-null float64 11 atemp 17379 non-null float64 12 hum 17379 non-null float64 13 windspeed 17379 non-null float64 14 casual 17379 non-null int64 15 registered 17379 non-null int64 16 cnt 17379 non-null int64 dtypes: float64(4), int64(12), object(1) memory usage: 2.3+ MB
# drop None
df =df.dropna()
# drop duplicates
df = df.drop_duplicates()
# dropping Instant and yr column as it is just a row identifier and counts fields
df = df.drop(['instant'], axis=1)
df = df.drop(['yr'], axis=1)
df = df.drop(['casual'], axis=1)
df = df.drop(['registered'], axis=1)
# adding new variables from existing column dteday : day
df['day'] = pd.DatetimeIndex(df['dteday']).day
# drop the dte day field
df.drop('dteday', 1, inplace=True)
# multiplying the temp ,atemp, hum, and windspeed by 100 to scale them
df['temp']=df['temp']*100
df['atemp']=df['atemp']*100
df['hum']=df['hum']*100
df['windspeed']=df['windspeed']*100
# adding new boolean columns
df['IsSummer']= (df['season']==3)
df['IsWinter']= (df['season']==1)
df['BadWeather']= (df['weathersit']>2)
df['IsWeekend']= (df['weekday']==0) | (df['weekday']==6)
# drop the weekday field
df.drop('weekday', 1, inplace=True)
C:\Users\Venkat\AppData\Local\Temp\ipykernel_15828\4246728557.py:17: FutureWarning: In a future version of pandas all arguments of DataFrame.drop except for the argument 'labels' will be keyword-only.
df.drop('dteday', 1, inplace=True)
C:\Users\Venkat\AppData\Local\Temp\ipykernel_15828\4246728557.py:32: FutureWarning: In a future version of pandas all arguments of DataFrame.drop except for the argument 'labels' will be keyword-only.
df.drop('weekday', 1, inplace=True)
df.describe()
| season | mnth | hr | holiday | workingday | weathersit | temp | atemp | hum | windspeed | cnt | day | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| count | 17379.000000 | 17379.000000 | 17379.000000 | 17379.000000 | 17379.000000 | 17379.000000 | 17379.000000 | 17379.000000 | 17379.000000 | 17379.000000 | 17379.000000 | 17379.000000 |
| mean | 2.501640 | 6.537775 | 11.546752 | 0.028770 | 0.682721 | 1.425283 | 49.698717 | 47.577510 | 62.722884 | 19.009761 | 189.463088 | 15.683411 |
| std | 1.106918 | 3.438776 | 6.914405 | 0.167165 | 0.465431 | 0.639357 | 19.255612 | 17.185022 | 19.292983 | 12.234023 | 181.387599 | 8.789373 |
| min | 1.000000 | 1.000000 | 0.000000 | 0.000000 | 0.000000 | 1.000000 | 2.000000 | 0.000000 | 0.000000 | 0.000000 | 1.000000 | 1.000000 |
| 25% | 2.000000 | 4.000000 | 6.000000 | 0.000000 | 0.000000 | 1.000000 | 34.000000 | 33.330000 | 48.000000 | 10.450000 | 40.000000 | 8.000000 |
| 50% | 3.000000 | 7.000000 | 12.000000 | 0.000000 | 1.000000 | 1.000000 | 50.000000 | 48.480000 | 63.000000 | 19.400000 | 142.000000 | 16.000000 |
| 75% | 3.000000 | 10.000000 | 18.000000 | 0.000000 | 1.000000 | 2.000000 | 66.000000 | 62.120000 | 78.000000 | 25.370000 | 281.000000 | 23.000000 |
| max | 4.000000 | 12.000000 | 23.000000 | 1.000000 | 1.000000 | 4.000000 | 100.000000 | 100.000000 | 100.000000 | 85.070000 | 977.000000 | 31.000000 |
df.info()
<class 'pandas.core.frame.DataFrame'> Int64Index: 17379 entries, 0 to 17378 Data columns (total 16 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 season 17379 non-null int64 1 mnth 17379 non-null int64 2 hr 17379 non-null int64 3 holiday 17379 non-null int64 4 workingday 17379 non-null int64 5 weathersit 17379 non-null int64 6 temp 17379 non-null float64 7 atemp 17379 non-null float64 8 hum 17379 non-null float64 9 windspeed 17379 non-null float64 10 cnt 17379 non-null int64 11 day 17379 non-null int64 12 IsSummer 17379 non-null bool 13 IsWinter 17379 non-null bool 14 BadWeather 17379 non-null bool 15 IsWeekend 17379 non-null bool dtypes: bool(4), float64(4), int64(8) memory usage: 1.8 MB
df = h2o.H2OFrame(df)
# Reponse column
y = "cnt"
Parse progress: |████████████████████████████████████████████████████████████████| (done) 100%
# Split into train & test
splits = df.split_frame(ratios = [0.8], seed = 1)
train = splits[0]
test = splits[1]
# Run AutoML for 1 minute
aml = H2OAutoML(max_runtime_secs=60, seed=1)
aml.train(y=y, training_frame=train)
AutoML progress: |█ 22:19:25.645: AutoML: XGBoost is not available; skipping it. 22:19:25.646: Step 'best_of_family_xgboost' not defined in provider 'StackedEnsemble': skipping it. 22:19:25.647: Step 'all_xgboost' not defined in provider 'StackedEnsemble': skipping it. ██████████████████████████████████████████████████████████████| (done) 100% Model Details ============= H2OStackedEnsembleEstimator : Stacked Ensemble Model Key: StackedEnsemble_AllModels_1_AutoML_5_20220306_221925 No model summary for this model ModelMetricsRegressionGLM: stackedensemble ** Reported on train data. ** MSE: 1030.841120734402 RMSE: 32.10671457397038 MAE: 21.299733452897698 RMSLE: NaN R^2: 0.9688112965309604 Mean Residual Deviance: 1030.841120734402 Null degrees of freedom: 10026 Residual degrees of freedom: 10022 Null deviance: 331409862.9163857 Residual deviance: 10336243.917603849 AIC: 98036.02656529291 ModelMetricsRegressionGLM: stackedensemble ** Reported on cross-validation data. ** MSE: 3267.6658207702208 RMSE: 57.16350077427222 MAE: 37.419970123986694 RMSLE: NaN R^2: 0.9021007511808254 Mean Residual Deviance: 3267.6658207702208 Null degrees of freedom: 13963 Residual degrees of freedom: 13959 Null deviance: 466158525.07057345 Residual deviance: 45629685.52123536 AIC: 152634.4461305511
# Explain leader model & compare with all AutoML models
exa = aml.explain(test)
Leaderboard shows models with their metrics. When provided with H2OAutoML object, the leaderboard shows 5-fold cross-validated metrics by default (depending on the H2OAutoML settings), otherwise it shows metrics computed on the frame. At most 20 models are shown by default.
| model_id | mean_residual_deviance | rmse | mse | mae | rmsle | training_time_ms | predict_time_per_row_ms | algo |
|---|---|---|---|---|---|---|---|---|
| StackedEnsemble_AllModels_1_AutoML_5_20220306_221925 | 3267.67 | 57.1635 | 3267.67 | 37.42 | nan | 520 | 0.035227 | StackedEnsemble |
| StackedEnsemble_AllModels_2_AutoML_5_20220306_221925 | 3270.2 | 57.1857 | 3270.2 | 37.4376 | nan | 500 | 0.042155 | StackedEnsemble |
| StackedEnsemble_BestOfFamily_3_AutoML_5_20220306_221925 | 3340.01 | 57.7929 | 3340.01 | 38.1735 | nan | 483 | 0.031644 | StackedEnsemble |
| StackedEnsemble_BestOfFamily_2_AutoML_5_20220306_221925 | 3340.75 | 57.7993 | 3340.75 | 38.1798 | nan | 452 | 0.031307 | StackedEnsemble |
| GBM_3_AutoML_5_20220306_221925 | 3356.91 | 57.9388 | 3356.91 | 38.4177 | nan | 1064 | 0.030055 | GBM |
| GBM_4_AutoML_5_20220306_221925 | 3386.16 | 58.1907 | 3386.16 | 37.9812 | nan | 1139 | 0.026926 | GBM |
| GBM_2_AutoML_5_20220306_221925 | 3595.1 | 59.9592 | 3595.1 | 40.0286 | nan | 901 | 0.024981 | GBM |
| GBM_5_AutoML_5_20220306_221925 | 3793.44 | 61.5909 | 3793.44 | 41.555 | nan | 775 | 0.023318 | GBM |
| StackedEnsemble_BestOfFamily_1_AutoML_5_20220306_221925 | 3810.33 | 61.7279 | 3810.33 | 41.7574 | nan | 489 | 0.033813 | StackedEnsemble |
| GBM_1_AutoML_5_20220306_221925 | 3811.23 | 61.7352 | 3811.23 | 41.7434 | nan | 1276 | 0.033502 | GBM |
| GBM_grid_1_AutoML_5_20220306_221925_model_1 | 4333.11 | 65.8263 | 4333.11 | 44.992 | nan | 780 | 0.02088 | GBM |
| DRF_1_AutoML_5_20220306_221925 | 4419.64 | 66.4804 | 4419.64 | 43.4485 | 0.442421 | 1926 | 0.01215 | DRF |
| GBM_grid_1_AutoML_5_20220306_221925_model_2 | 4774.18 | 69.0954 | 4774.18 | 47.7127 | nan | 589 | 0.016533 | GBM |
| XRT_1_AutoML_5_20220306_221925 | 5867.56 | 76.6 | 5867.56 | 49.7583 | 0.5222 | 1181 | 0.006752 | DRF |
| GBM_grid_1_AutoML_5_20220306_221925_model_3 | 6105.05 | 78.1348 | 6105.05 | 52.6356 | nan | 165 | 0.004975 | GBM |
| DeepLearning_1_AutoML_5_20220306_221925 | 13038 | 114.184 | 13038 | 73.527 | nan | 433 | 0.001543 | DeepLearning |
| GBM_grid_1_AutoML_5_20220306_221925_model_4 | 18872.3 | 137.376 | 18872.3 | 105.372 | 1.35474 | 47 | 0.000899 | GBM |
| DeepLearning_grid_1_AutoML_5_20220306_221925_model_1 | 19913.7 | 141.116 | 19913.7 | 99.0284 | nan | 1074 | 0.003413 | DeepLearning |
| GLM_1_AutoML_5_20220306_221925 | 21745.6 | 147.464 | 21745.6 | 108.853 | nan | 194 | 0.000293 | GLM |
Residual Analysis plots the fitted values vs residuals on a test dataset. Ideally, residuals should be randomly distributed. Patterns in this plot can indicate potential problems with the model selection, e.g., using simpler model than necessary, not accounting for heteroscedasticity, autocorrelation, etc. Note that if you see "striped" lines of residuals, that is an artifact of having an integer valued (vs a real valued) response variable.
The variable importance plot shows the relative importance of the most important variables in the model.
Variable importance heatmap shows variable importance across multiple models. Some models in H2O return variable importance for one-hot (binary indicator) encoded versions of categorical columns (e.g. Deep Learning, XGBoost). In order for the variable importance of categorical columns to be compared across all model types we compute a summarization of the the variable importance across all one-hot encoded features and return a single variable importance for the original categorical feature. By default, the models and variables are ordered by their similarity.
This plot shows the correlation between the predictions of the models. For classification, frequency of identical predictions is used. By default, models are ordered by their similarity (as computed by hierarchical clustering). Interpretable models, such as GAM, GLM, and RuleFit are highlighted using red colored text.
SHAP summary plot shows the contribution of the features for each instance (row of data). The sum of the feature contributions and the bias term is equal to the raw prediction of the model, i.e., prediction before applying inverse link function.
Partial dependence plot (PDP) gives a graphical depiction of the marginal effect of a variable on the response. The effect of a variable is measured in change in the mean response. PDP assumes independence between the feature for which is the PDP computed and the rest.
An Individual Conditional Expectation (ICE) plot gives a graphical depiction of the marginal effect of a variable on the response. ICE plots are similar to partial dependence plots (PDP); PDP shows the average effect of a feature while ICE plot shows the effect for a single instance. This function will plot the effect for each decile. In contrast to the PDP, ICE plots can provide more insight, especially when there is stronger feature interaction.
# Explain a single H2O model (e.g. leader model from AutoML)
exm = aml.leader.explain(test)
Residual Analysis plots the fitted values vs residuals on a test dataset. Ideally, residuals should be randomly distributed. Patterns in this plot can indicate potential problems with the model selection, e.g., using simpler model than necessary, not accounting for heteroscedasticity, autocorrelation, etc. Note that if you see "striped" lines of residuals, that is an artifact of having an integer valued (vs a real valued) response variable.
Partial dependence plot (PDP) gives a graphical depiction of the marginal effect of a variable on the response. The effect of a variable is measured in change in the mean response. PDP assumes independence between the feature for which is the PDP computed and the rest.
An Individual Conditional Expectation (ICE) plot gives a graphical depiction of the marginal effect of a variable on the response. ICE plots are similar to partial dependence plots (PDP); PDP shows the average effect of a feature while ICE plot shows the effect for a single instance. This function will plot the effect for each decile. In contrast to the PDP, ICE plots can provide more insight, especially when there is stronger feature interaction.
On the x-axis is the SHAP value. Indicates how much is the change in log-odds. From this number we can extract the probability of success.
We can see when the weather is bad, there are less bike rentals. This is seen from High and Negative SHAP values
We can also see the when the windspeed is high the bike rentals are less. This is intuitive and could be seen from High and Negative SHAP values.
When Temperature is slightly higher, there are more bike rentals. But as soon as temp decreases, there are sharp reduction of bike rentals